Return on Investment Results (Part 3)

Microsoft’s Public Sector Center for Digital Skills recently collaborated with the ROI Institute and used their methodology to embark on a comprehensive evaluation study with a government customer to assess the Return on Investment (ROI) of their skilling programs.

This article presents the findings from a comprehensive evaluation study conducted by the ROI Institute on the Procurement for Practitioners Program, a 2-day training initiative designed specifically for procurement professionals in the public sector who are navigating the digital era.  

The evaluation, which assessed 171 participants across nine cohorts over the past two years, found that attendees not only responded favorably to the program but also exhibited a notable 12.5% increase in job proficiency. Follow-up questionnaires revealed that 19% of participants applied the skills they learned at least once a week, with 68% reporting success in the application of these skills. 

The study revealed the financial impact of the program generated a total monetary benefit of $220,915 compared to a fully-loaded cost of $155,000, resulting in a ROI of 43%. In addition to these quantifiable benefits, the program also delivered intangible advantages such as a reduction in contract variations and enhanced adherence to procurement principles. 

Let’s explore the steps the ROI Institute took to determine this positive contribution and learn how the methodology employed by the ROI Institute was used to calculate these results and to learn how these insights can be leveraged to optimize your own skilling initiatives within your public sector organization. 

Evaluation study methodology in action  

To start defining the ROI of the Procurement Practitioners Program, the ROI Institute followed the four phases of their ROI process model.  

Phase 1: Plan the Evaluation  

Planning began as soon as it was decided that an impact/ROI study would be conducted. Since the program was already active within the organization, steps were taken to document the program objectives and validate it’s alignment to the business. 

Once the alignment to the business was validated, comprehensive data collection and ROI analysis plans were developed. The data collection plan outlined: 

• The objectives for each evaluation level 
• The method for collecting the data 
• When the data would be collected 
• Who was responsible for collecting it 
• The methods used for isolating the effects of the program 
• How the measure(s) would be converted to monetary value 
• The program’s cost categories 
• Targeted intangible benefits 
• The methods for communicating the results 

Phase 2: Collect Data: 

Once the planning documents were finalized, data collection started. Methods for collecting data in this study included knowledge/skill checks, end-of-program survey, follow-up questionnaire and review of business records. 

A review of existing evaluation data provided insights into the reaction and learning that occurred while a comprehensive questionnaire was developed to capture the extent to which participants applied what they learned and the resulting impact to the business.  

When using a survey or questionnaire to collect data, you should develop an administration plan that provides details regarding who it will be sent to, why it is being used, what data is being collected, how it will be distributed, and when it will occur.  This plan will help ensure the needed data is collected for a successful evaluation.  

Phase 3: Analyze Data 

Once all the data was collected, the analysis phase began. Critical steps were completed to determine the impact of the program, including: 

Isolating Program Effects: This step identifies the specific contributions of the program to be used when calculating the ROI. Isolation techniques include control group, trend line analysis, and the estimation process which was used for this study. Program participants were asked to estimate the extent to which the program contributed to the reported improvement in a measure, along with their confidence level in their estimates. 

Converting Data to Monetary Value: To calculate the return on investment, business impact data collected is converted to a monetary amount. This requires a value to be placed on each unit of a measure connected with the program. For this study, standard values were used. 

Tabulating Program Costs: Tabulating the costs involves identifying all of the related costs of the program. 

Please note you should follow a conservative approach when converting an improvement to monetary value and tabulating the program costs.  For program benefits, when in doubt, leave it out and report as an intangible benefit.  For program costs, when in doubt, leave it in, using an estimated cost if needed. 

Phase 4: Optimize Results  

The final phase in the evaluation involves communicating the findings and using the results to make applicable improvements to the program. Sharing the results with stakeholders, program owners, participants, and other audiences should occur during and after the program’s completion. Additionally, with the competition for resources, communicating results and identifying improvement opportunities helps secure funding and foster future success of the program.  

Conclusion  

• Measuring the return on investment (ROI) of public sector skilling programs demonstrates the value of the program and how it contributes to an organization’s success.
• With a clear business case for skilling and robust ROI evaluations, you can defend critical skilling budgets from cuts and improve program and employee performance.  
• Determining the ROI in skilling is not just a measure of financial return but a crucial step in fostering a competent and resilient government workforce. 

References 

Jack J. Phillips, and Patti P. Phillips, Handbook of Training Evaluation and Measurement Methods, 4th edition. New York, NY: Routledge (2016). 

Patricia Pulliam Phillips, Jack J. Phillips, Gina Paone, and Cyndi Huff Gaudet, Value for Money: How to Show the Value for Money for All Types of Projects and Programs. Beverly, MA: Scrivener Publishing (2019). 

• The Importance of Defining the Return on Investment for Skilling within Public Sector Organizations (Part 1) 
• Best Practices to Help You Define the Return on Investment for Skilling within Public Sector Organizations (Part 2) 
• ROI Methodology Reference Material: Supplemental resources for learning more about conducting evaluations and building capability to evaluate your programs.